Saliency-weighted video quality assessment

ABSTRACT

Systems and methods are disclosed for weighting the image quality prediction of any visual-attention-agnostic quality metric with a saliency map. By accounting for the salient regions of an image or video frame, the disclosed systems and methods may dramatically improve the precision of the visual-attention-agnostic quality metric during image or video quality assessment. In one implementation, a method of saliency-weighted video quality assessment includes: determining a per-pixel image quality vector of an encoded video frame; determining per-pixel saliency values of the encoded video frame or a reference video frame corresponding to the encoded video frame; and computing a saliency-weighted image quality metric of the encoded video frame by weighting the per-pixel image quality vector using the per-pixel saliency values.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of and claims priority to U.S. patentapplication Ser. No. 14/954,652 filed on Nov. 30, 2015 and titled“SALIENCY-WEIGHTED VIDEO QUALITY ASSESSMENT.”

TECHNICAL FIELD

The present disclosure relates generally to video quality assessmenttechniques, and more particularly, some embodiments relate to systemsand methods for providing saliency-weighted video quality assessment.

DESCRIPTION OF THE RELATED ART

The goal of automated video quality assessment is obtaining aquantitative measure that is highly correlated with the perceivedquality of the input visual content. Currently, video quality assessmentand control is mostly done interactively by quality experts, who have toinspect the produced (e.g., transcoded) versions of the evaluatedcontent and make sure that they conform to corresponding qualityrequirements. Automated video quality assessment tools are highlydesired in content distribution workflows due to their potential tosignificantly reduce the amount of manual work required during thequality control process. However, automated methods can only be trustedif they consistently provide quality predictions that are correlatedwith the subjectively perceived content quality.

BRIEF SUMMARY OF THE DISCLOSURE

According to various embodiments of the technology disclosed herein,systems and methods are disclosed for providing saliency-weighted videoquality assessment. In one embodiment, an assessed image is received ata non-transitory computer readable medium. In this embodiment, one ormore processors: determine a per-pixel image quality vector of theassessed image; determine per-pixel saliency values of the assessedimage or a reference image corresponding to the assessed image; andcompute a saliency-weighted image quality metric of the assessed imageby weighting the per-pixel image quality vector using the per-pixelsaliency values. In various implementations of this embodiment, theassessed image is an encoded video frame corresponding to a video, andthe steps of determining a per-pixel image quality vector, determiningper-pixel saliency values, and computing a saliency-weighted imagequality metric are repeated for a plurality of encoded video frames thatcorrespond to the video.

In some embodiments, the per-pixel image quality vector of the assessedimage is determined by comparing the assessed image with a correspondingreference image. In alternative embodiments, the per-pixel image qualityvector of the assessed image is determined using a blind qualityassessment method or a reduced-reference quality assessment method.

In some embodiments, a graphical user interface (GUI) may be providedfor performing saliency-weighted video quality assessment. In theseembodiments, a saliency-weighted image quality metric may be determinedfor a plurality of encoded video frames corresponding to a video, andthe GUI may display a time plot of the saliency-weighted image qualitymetric for each of the encoded video frames corresponding to the video.In yet further embodiments, the GUI may display a visual comparison ofeach of the encoded video frames and corresponding reference videoframes that are not encoded.

In one embodiment, a method of video quality assessment includes:receiving an encoded video; determining a saliency-weighted imagequality metric for each video frame of the encoded video; determiningvideo frames of the encoded video with a saliency-weighted image qualitymetric that is below a threshold; and assembling a playlist includingthe video frames of the encoded video with a saliency-weighted imagequality metric that is below the threshold.

Other features and aspects of the disclosed method will become apparentfrom the following detailed description, taken in conjunction with theaccompanying drawings, which illustrate, by way of example, the featuresin accordance with embodiments of the disclosure. The summary is notintended to limit the scope of the claimed disclosure, which is definedsolely by the claims attached hereto.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

The present disclosure, in accordance with one or more variousembodiments, is described in detail with reference to the followingfigures. The figures are provided for purposes of illustration only andmerely depict typical or example embodiments of the disclosure.

FIG. 1 illustrates an example environment in which saliency-weightedvideo quality assessment may take place in accordance with the presentdisclosure.

FIG. 2 is an operational flow diagram illustrating a method ofsaliency-weighted image quality assessment in accordance with thepresent disclosure.

FIG. 3 is an operational flow diagram illustrating an example method ofcomputing a saliency map for an image in accordance with the presentdisclosure.

FIG. 4A illustrates an example saliency-weighted video qualityassessment graphical user interface that may be used in accordance withthe present disclosure.

FIG. 4B illustrates an encoded video frame, a corresponding saliencymap, and visual differences between the encoded and unencoded videoframe that may be displayed by the GUI of FIG. 4A.

FIG. 5 is an operational flow diagram illustrating an example method ofusing the saliency-weighted image quality assessment method of FIG. 2 toautomate and facilitate the quality control assessment of an encodedvideo in accordance with embodiments of the disclosure.

FIG. 6 illustrates an example computing module that may be used toimplement various features of the saliency-weighted video qualityassessment systems and methods disclosed herein.

The figures are not exhaustive and do not limit the disclosure to theprecise form disclosed.

DETAILED DESCRIPTION

An overlooked but highly influential aspect of human vision in videoquality assessment is visual attention. Each video sequence (even moreso in cinematic content) often has one or more salient elements that arelikely to catch the viewer's attention. As a result, most viewers' gazeis fixated in a subset of the content presented on the display. A visualartifact in a salient image region is subjectively more disturbing thananother artifact with the same magnitude but located in a non-salientregion.

In accordance with embodiments of the technology disclosed herein,systems and methods are disclosed for weighting the image (e.g., videoframe) quality prediction of any visual-attention-agnostic qualitymetric with a saliency map. By accounting for the salient regions of theimage, the disclosed system and method may dramatically improve theprecision of the visual-attention-agnostic quality metric. In aparticular embodiment, the visual-attention-agnostic quality metric isthe Structural Similarity (SSIM) index metric and the saliency map isdetermined by applying a contrast-based saliency estimation method.

In some embodiments, the saliency-weighted video quality assessmentmethod may be implemented using an application including a graphicaluser interface (GUI). In these embodiments, the video quality assessmentmay be performed fully automatically, and the results can be stored,displayed and communicated in various ways, to assist users in a varietyof different quality assessment applications. In a particularimplementation of this embodiment, the application may assemble aplaylist of video frames that fall below a particular quality threshold(as determined by the saliency-weighted video quality assessment method)for playback by the GUI for further quality assessment.

Before describing the invention in detail, it is useful to describe anexample environment in which the invention can be implemented. FIG. 1illustrates one such example environment 100.

In environment 100, an assessed video 121 (e.g., encoded video) isreceived for objective saliency-weighted video quality assessment (step170) by saliency-weighted video quality assessment system 102. Theassessed video 121 may include a film, a trailer, an episode from aseries, a commercial, a video game cutscene, and the like. In someembodiments, illustrated by FIG. 1, the video quality assessment ofvideo 121 may be conducted using a full reference method, i.e., bycomparing video 121 to an uncompressed and distortion-free (e.g.,unencoded) reference video 122. In alternative embodiments (e.g., wherea full quality reference video 122 is unavailable), the video qualityassessment method may be conducted using a blind quality assessmentmethod (no reference video available) or a reduced-reference qualityassessment method (some information relating to the full quality videois available for comparison). As a result of the saliency-weighted videoquality assessment step 170, an objective metric of video quality (e.g.,a normalized number metric between 0 and 1) may be provided for eachframe of the assessed video 121, video 121 as a whole, or somecombination thereof.

Following video quality assessment of video 121 by system 170, anynumber of quality control related video processing applications (step180) may be implemented. In one embodiment, the quality of assessedvideo 121 may be adjusted (application 181). For example, a contentserver transmitting assessed video 121 over a computer network mayexamine the objective quality metric of video 121 being transmitted andmake video bitrate and other appropriate adjustments. As another exampleof application 181, quality control operators may be notified if qualitythresholds are missed. In another embodiment, the objective qualitymetric may be used to optimize video processing applications used tocreate assessed video 121 (application 182). For example, a user, anautomated software application, or some combination thereof may adjustalgorithms and parameter settings (e.g., transcoding parameters) of avideo processing system or application used to create assessed video121. In yet another embodiment, the objective quality metric may be usedto benchmark a video processing system or application used to createassessed video 121 (application 183).

As would be appreciated by one having skill in the art, any number ofvideo processing applications 180 may be implemented following videoquality assessment 170. Moreover, steps 170 and 180 may be recursivelyrepeated to assist in applications such as monitoring and adjusting thequality of assessed video 121, optimizing a video processingapplication, benchmarking a video processing application, and the like.

Referring back to saliency-weighted video quality assessment system 102,in various embodiments, system 102 may be any computing system(workstation, laptop, smartphone, etc.) configured to receive a video121 or image and determine an objective video quality or image qualitymetric using a saliency-weighted assessment method. As illustrated,system 102 includes a connectivity interface 131, storage 132 forstoring a saliency-weighted video quality assessment application 133 andassessed videos 121 or images, processor 134, and one or more displays135.

Processor 134 executes a saliency-weighted video quality assessmentapplication 133 that determines one or more saliency-weighted metricsfor objectively assessing the quality of an input video 121 whencompared with a full quality reference video 122. In some embodiments,the objective metric of video quality may be provided for each frame ofthe assessed video 121, video 121 as a whole, or some combinationthereof.

In some embodiments, further described below, application 133 provides aGUI for a user (e.g., quality control specialist) to visualize (e.g.,using display 135) and evaluate the assessed video using thesaliency-weighted video quality metrics. For example, for an assessedvideo 121, application 133 may display a plot showing thesaliency-weighted quality of each video frame as a function of itssequence in the video's time code, a determined saliency map for eachassessed video frame, a visual comparison between the assessed video 121and reference video 122, and other video quality assessment tools. Inthese embodiments, application 133 may be integrated as part of a videoediting application, an image editing application, a video game designapplication, or some combination thereof. In this manner, a qualitycontrol specialist may immediately take corrective action based on theassessed quality of the video.

In one embodiment, application 133 may be integrated as part of acontent creation and distribution application that distributes theassessed video 121. For example, if the quality of assessed video 121falls below a certain quality threshold, the application mayautomatically adjust transcoding parameters to create and distribute anew video 121 that meets the quality threshold. In these embodiments,connectivity interface 131 may connect system 102 to a contentdistribution network using a wired or wireless network connection suchas a local area network connection, a cellular network connection, asatellite network connection, or the like.

Although example environment 100 was primarily described with respect tothe saliency-weighted assessment of video quality for use in videoprocessing applications, it should be noted that in other embodimentsthe invention may be implemented in an environment that focuses on thesaliency-weighted assessment of image quality for use in imageprocessing applications.

FIG. 2 is an operational flow diagram illustrating a method 200 ofsaliency-weighted image quality assessment in accordance with thepresent disclosure. Method 200 takes an assessed image 210 (e.g., avideo frame) and full quality reference image 220 as inputs, and outputsa saliency-weighted image quality metric for the assessed image 210. Asfurther described below, the saliency-weighted image quality metric isdetermined by weighting an image quality estimate vector of the image210 using a saliency map of the image. In various embodiments, some orall of the process operations of method 200 (e.g., determination ofimage quality vector, determination of saliency map for image, anddetermination of saliency-weighted image quality metric) may beimplemented by system 102 using application 133.

Prior to beginning method 200, system 102 receives an assessed image 210(e.g., encoded image) and a comparison full quality reference image 220(e.g., unencoded, raw image) for the saliency-weighted image qualityassessment. In some embodiments, various preprocessing operations (notshown) may be performed prior to beginning method 200 to make assessedimage 210 and reference image 220 suitable for comparison. For example,application 133 may be used to align and scale the images, transform thecolor space of the images, gamma-correct the images, and otherwisefilter the images in preparation for comparison.

At operation 230, a per-pixel image quality estimate vector isdetermined for the assessed image 210 by comparing assessed image 210with reference image 220. The per-pixel image quality estimate providesa per-pixel numerical estimate of the quality of assessed image 210. Forexample, the per-pixel quality estimate vector may provide an estimateof the quality of each pixel of assessed image 210 as compared withreference image 220.

In one particular embodiment, the per-pixel image quality vector forassessed image 210 is determined using the structural similarity (SSIM)index metric that numerically defines the similarity between assessedimage 210 and full quality reference image 220 based on a comparison ofthe luminance, contrast, and structure of pixels of assessed image 210and reference image 220. In one embodiment, the SSIM may bemathematically defined by Equation (1):

SSIM(I,J)=l(μ_(I), μ_(J))^(α) c(σ_(I),μ_(J))^(β) s(σ_(I),σ_(J))^(γ)  (1)

Where I and J are gamma-corrected input images both having N pixels,comparison functions l, c, and s and denote the luminance, contrast andstructure terms, respectively, μ and σ denote the mean and standarddeviation, respectively, and α>0, β>0, and γ>0 are model parameters thatcontrol the weights of the three terms of the metric.

In some embodiments, the SSIM metric may be implemented as a multi-scalestructural similarity (MS-SSIM) metric. In other embodiments, other fullreference metrics such as a visual signal-to-noise ratio (VSNR) metric,peak signal-to-noise ratio (PSNR) metric or mean squared error (MSE)metric may be used for determining the per-pixel image quality vector ofassessed image 210. In alternative embodiments, where no reference image220 is available, blind quality assessment methods or reduced-referencequality assessment methods may be implemented to determine the per-pixelimage quality vector. For example, a no-reference image quality metricmay be implemented as described in Robert Herzog, Martin C̆adík, Tunç O.Aydčin, Kwang In Kim, Karol Myszkowski, Hans-P. Seidel, NoRM:No-Reference Image Quality Metric for Realistic Image Synthesis,Computer Graphics Forum, v. 31 n.2pt4, p. 545-554, May 2012.

At operation 240, a saliency map with per-pixel image saliency values isdetermined for the assessed image 210. The saliency map identifies imageelements that are likely to catch the attention of a human observer. Insome embodiments, the saliency map may be determined based on variousfeatures of assessed image 210, such as, for example, color variation ofindividual pixels, edges and gradients, spatial frequencies, structureand distribution of image patches, histograms, or some combinationthereof. In particular embodiments, the saliency map may be determinedby applying a contrast-based saliency estimation method.

FIG. 3 illustrates one such example of a contrast-based saliencyestimation method 240 in accordance with embodiments of the technologydisclosed herein. The example method of FIG. 3 begins at operation 241,during which the assessed image 210 is abstracted into image elementsthat preserve relevant structure while removing undesirable details. Forexample, an adaptation of the simple linear iterative clustering (SLIC)super-pixels process may be used to abstract the image into perceptuallyhomogenous or uniform regions. As used herein to describe an image, theterm “element” may be defined as one pixel or a group of pixels havingsimilar features. For a group of pixels having similar features, thefeatures may be the pixels' values or any other features that may becalculated out of the pixels' values, such as features measuring color,texture, disparity or motion. At operation 242, the uniqueness of imageelements (i.e., rarity of each element given its position in the imageand discriminating feature) is determined. At operation 243, the spatialdistribution of the elements is calculated. For example, the spatialvariance of an element's color (i.e., occurrence elsewhere in the image)may be determined.

Following determination of the uniqueness and spatial distribution ofimage elements, at operation 244, these two measures are combined tocompute a per-element saliency measure or value. Once per elementsaliency is computed, at operation 245 the per-pixel saliency values maybe derived, thus producing the saliency map. In one embodiment, theper-pixel saliency values may be derived from the per-element saliencyvalues by up-sampling the per-element saliency values with a Gaussianweight. A particular mathematical implementation of method 240 isdescribed in U.S. Pat. No. 9,025,880, titled “Visual saliency estimationfor images and video” and issued May 5, 2015, which is hereinincorporated by reference in its entirety.

It should be noted that although the determination of the saliency map(operation 240) in the example of FIG. 2 is illustrated as being done onassessed image 210, in alternative embodiments, the saliency map mayinstead be computed from reference image 220 using the same saliency mapdetermination method.

Referring back to method 200, following determination of the saliencymap with per-pixel image saliency values (operation 240) and a per-pixelimage quality estimate vector (operation 230), at operation 250 asaliency-weighted image quality metric is determined by weighting thesaliency agnostic (i.e., visual-attention-agnostic) per-pixel imagequality estimate vector using the per-pixel saliency values of thesaliency map. As would be appreciated by one having skill in the art,any visual-attention-agnostic metric that generates per-pixel imagequality values may be weighted with a saliency map using the disclosedmethod. By accounting for the salient regions of the image, thedisclosed system and method may dramatically improve the precision ofthe visual-attention-agnostic quality metric.

In one mathematical implementation, the saliency-weighted image qualitymetric may be defined by Equation (2):

$\begin{matrix}{\hat{q} = {\frac{1}{N}\left( {q \cdot s^{w}} \right)}} & (2)\end{matrix}$

Where {circumflex over (q)} is a scalar quality index, (·) denotes theinner product, q is a per-pixel quality estimate vector, which in oneembodiment can be obtained using SSIM, vector s contains thecorresponding per-pixel saliency values, w is the weight given to thesaliency map, and N is the number of pixels in the image.

In various embodiments, method 200 may be used to conduct asaliency-weighted video quality assessment to determine a single qualitymetric for a video. In such embodiments, method 200 may be repeated foreach video frame to compute a quality metric for each video frame, andthese individual per-frame quality metrics may be averaged over theentire video sequence. In implementations of these embodiments, thevideo frames may be weighted prior to averaging.

Following determination of the saliency-weighted image quality metric atoperation 250, at optional operation 260, additional image (or video)processing may be conducted based on the determined metric. For example,the quality metric may be used to monitor and adjust image or videoquality, optimize image or video processing applications used to createthe image or video, benchmark image or video processing systems orapplication used to create the image or video, and perform other likeoperations.

In some embodiments, a video quality control specialist, video editor,or other user may utilize an interactive GUI provided bysaliency-weighted video quality assessment application 133 to assist invideo quality assessment. FIGS. 4A-4B illustrate one particularimplementation of a GUI 300 that may be used by a user of system 102 toassess video quality using method 200.

As illustrated in this particular embodiment, interface 300 provides adisplay 330 that allows a user to watch the video being assessed (e.g.,encoded video) or unencoded reference video, and a display 340 that mayconcurrently provide color coded (where lack of color denotes no visibleartifacts) per-pixel visible difference visualization between thereference video and encoded video based on a particular video qualityassessment method. GUI 300 additionally includes selectable controls320A-320C that enable a user to select the video quality assessmentmethod (e.g., saliency-weighted SSIM (control 320C), pure SSIM (control320A)) being visualized on display 340 or the saliency map (control320B) of the frame being displayed. For example, in this particularembodiment, the saliency-weighted SSIM control is selected, ashighlighted by the color-coded focus on the sitting character of thevideo.

Additionally, interface 300 displays a per-frame quality prediction plot350 over the entire video sequence. As illustrated in this particularembodiment, plot 350 provides the video quality normalized to a scale(γ-axis) as a function of time code or video frame number (x-axis) ofthe video. In some embodiments, comparison time plots may be providedfor different video quality assessment methods. For example, time plots350 may be displayed for both saliency-weighted SSIM and pure SSIM videoquality assessment methods.

FIG. 4B illustrates a video frame 360 that may be displayed by display330 using GUI 300, and a corresponding saliency map 370, pure SSIMvisual differences 380, and saliency-weighted SSIM visual differences390 that may be displayed by display 340 of GUI 300. In this example,red denotes highly visible artifacts, green denotes less visibleartifacts, and the absence of color denotes no visible artifacts. Asillustrated, the output 380 of SSIM without saliency weighting fails toaccount for the fact that the majority of the viewers will be focusingon the person and therefore will be less sensitive to the visualartifacts in the background. By contrast, the output 390 of thesaliency-weighted SSIM method considers the viewer's focus on theperson, the most salient region of the video frame.

FIG. 5 is an operational flow diagram illustrating an example method 400of using saliency-weighted image quality assessment method 200 toautomate and facilitate the quality control assessment of an encodedvideo 401 in accordance with embodiments of the disclosure. In someembodiments, method 400 may be implemented by executing application 133,and method 400 may be visualized using GUI 300.

Method 400 begins at operation 420, where the saliency-weighted imagequality metric is determined for each video frame of encoded video 401.In some embodiments, this metric may be determined by repeating method200 for each video frame. At operation 430, the time codes or framenumbers of video frames with a saliency-weighted image quality metricthat falls below a certain threshold may be marked for further qualityassessment. In some embodiments, operation 430 may occur after orconcurrent with operation 420 (e.g., time codes may be marked on an adhoc basis after a frame's weighted image quality is determined). Atoperation 440, the marked time codes (i.e., frames with below thresholdimage quality) may be automatically assembled into a playlist forfurther quality assessment. By way of example, consider a two-hourencoded video 401 that must exceed a normalized threshold quality of 0.9or 90%. If all but a two-minute segment of this video meets or exceeds athreshold quality of 0.9, method 400 may be used to create a playlistfor playing back the two-minute segment. In this manner, a video qualitycontrol specialist may focus remedial action on the problematictwo-minute segment.

FIG. 6 illustrates an example computing module that may be used toimplement various features of the systems and methods disclosed herein.As used herein, the term module might describe a given unit offunctionality that can be performed in accordance with one or moreembodiments of the present application. As used herein, a module mightbe implemented utilizing any form of hardware, software, or acombination thereof. For example, one or more processors, controllers,ASICs, PLAs, PALs, CPLDs, FPGAs, logical components, software routinesor other mechanisms might be implemented to make up a module. Inimplementation, the various modules described herein might beimplemented as discrete modules or the functions and features describedcan be shared in part or in total among one or more modules. In otherwords, as would be apparent to one of ordinary skill in the art afterreading this description, the various features and functionalitydescribed herein may be implemented in any given application and can beimplemented in one or more separate or shared modules in variouscombinations and permutations. Even though various features or elementsof functionality may be individually described or claimed as separatemodules, one of ordinary skill in the art will understand that thesefeatures and functionality can be shared among one or more commonsoftware and hardware elements, and such description shall not requireor imply that separate hardware or software components are used toimplement such features or functionality.

Where components or modules of the application are implemented in wholeor in part using software, in one embodiment, these software elementscan be implemented to operate with a computing or processing modulecapable of carrying out the functionality described with respectthereto. One such example computing module is shown in FIG. 6. Variousembodiments are described in terms of this example—computing module 600.After reading this description, it will become apparent to a personskilled in the relevant art how to implement the application using othercomputing modules or architectures.

Referring now to FIG. 6, computing module 600 may represent, forexample, computing or processing capabilities found within desktop,laptop, notebook, and tablet computers; hand-held computing devices(tablets, PDA's, smart phones, cell phones, palmtops, etc.); mainframes,supercomputers, workstations or servers; or any other type ofspecial-purpose or general-purpose computing devices as may be desirableor appropriate for a given application or environment. Computing module600 might also represent computing capabilities embedded within orotherwise available to a given device. For example, a computing modulemight be found in other electronic devices such as, for example, digitalcameras, navigation systems, cellular telephones, portable computingdevices, modems, routers, WAPs, terminals and other electronic devicesthat might include some form of processing capability.

Computing module 600 might include, for example, one or more processors,controllers, control modules, or other processing devices, such as aprocessor 604. Processor 604 might be implemented using ageneral-purpose or special-purpose processing engine such as, forexample, a microprocessor, controller, or other control logic. In theillustrated example, processor 604 is connected to a bus 602, althoughany communication medium can be used to facilitate interaction withother components of computing module 600 or to communicate externally.

Computing module 600 might also include one or more memory modules,simply referred to herein as main memory 608. For example, preferablyrandom access memory (RAM) or other dynamic memory, might be used forstoring information and instructions to be executed by processor 604.Main memory 608 might also be used for storing temporary variables orother intermediate information during execution of instructions to beexecuted by processor 604. Computing module 600 might likewise include aread only memory (“ROM”) or other static storage device coupled to bus602 for storing static information and instructions for processor 604.

The computing module 600 might also include one or more various forms ofinformation storage mechanism 610, which might include, for example, amedia drive 612 and a storage unit interface 620. The media drive 612might include a drive or other mechanism to support fixed or removablestorage media 614. For example, a hard disk drive, a solid state drive,a magnetic tape drive, an optical disk drive, a CD, DVD, or Blu-raydrive (R or RW), or other removable or fixed media drive might beprovided. Accordingly, storage media 614 might include, for example, ahard disk, a solid state drive, magnetic tape, cartridge, optical disk,a CD, DVD, Blu-ray or other fixed or removable medium that is read by,written to or accessed by media drive 612. As these examples illustrate,the storage media 614 can include a computer usable storage mediumhaving stored therein computer software or data.

In alternative embodiments, information storage mechanism 610 mightinclude other similar instrumentalities for allowing computer programsor other instructions or data to be loaded into computing module 600.Such instrumentalities might include, for example, a fixed or removablestorage unit 622 and an interface 620. Examples of such storage units622 and interfaces 620 can include a program cartridge and cartridgeinterface, a removable memory (for example, a flash memory or otherremovable memory module) and memory slot, a PCMCIA slot and card, andother fixed or removable storage units 622 and interfaces 620 that allowsoftware and data to be transferred from the storage unit 622 tocomputing module 600.

Computing module 600 might also include a communications interface 624.Communications interface 624 might be used to allow software and data tobe transferred between computing module 600 and external devices.Examples of communications interface 624 might include a modem orsoftmodem, a network interface (such as an Ethernet, network interfacecard, WiMedia, IEEE 802.XX or other interface), a communications port(such as for example, a USB port, IR port, RS232 port Bluetooth®interface, or other port), or other communications interface. Softwareand data transferred via communications interface 624 might typically becarried on signals, which can be electronic, electromagnetic (whichincludes optical) or other signals capable of being exchanged by a givencommunications interface 624. These signals might be provided tocommunications interface 624 via a channel 628. This channel 628 mightcarry signals and might be implemented using a wired or wirelesscommunication medium. Some examples of a channel might include a phoneline, a cellular link, an RF link, an optical link, a network interface,a local or wide area network, and other wired or wireless communicationschannels.

In this document, the terms “computer program medium” and “computerusable medium” are used to generally refer to transitory ornon-transitory media such as, for example, memory 608, storage unit 620,media 614, and channel 628. These and other various forms of computerprogram media or computer usable media may be involved in carrying oneor more sequences of one or more instructions to a processing device forexecution. Such instructions embodied on the medium, are generallyreferred to as “computer program code” or a “computer program product”(which may be grouped in the form of computer programs or othergroupings). When executed, such instructions might enable the computingmodule 600 to perform features or functions of the present applicationas discussed herein.

Although described above in terms of various exemplary embodiments andimplementations, it should be understood that the various features,aspects and functionality described in one or more of the individualembodiments are not limited in their applicability to the particularembodiment with which they are described, but instead can be applied,alone or in various combinations, to one or more of the otherembodiments of the application, whether or not such embodiments aredescribed and whether or not such features are presented as being a partof a described embodiment. Thus, the breadth and scope of the presentapplication should not be limited by any of the above-describedexemplary embodiments.

Terms and phrases used in this document, and variations thereof, unlessotherwise expressly stated, should be construed as open ended as opposedto limiting. As examples of the foregoing: the term “including” shouldbe read as meaning “including, without limitation” or the like; the term“example” is used to provide exemplary instances of the item indiscussion, not an exhaustive or limiting list thereof; the terms “a” or“an” should be read as meaning “at least one,” “one or more” or thelike; and adjectives such as “conventional,” “traditional,” “normal,”“standard,” “known” and terms of similar meaning should not be construedas limiting the item described to a given time period or to an itemavailable as of a given time, but instead should be read to encompassconventional, traditional, normal, or standard technologies that may beavailable or known now or at any time in the future. Likewise, wherethis document refers to technologies that would be apparent or known toone of ordinary skill in the art, such technologies encompass thoseapparent or known to the skilled artisan now or at any time in thefuture.

The presence of broadening words and phrases such as “one or more,” “atleast,” “but not limited to” or other like phrases in some instancesshall not be read to mean that the narrower case is intended or requiredin instances where such broadening phrases may be absent. The use of theterm “module” does not imply that the components or functionalitydescribed or claimed as part of the module are all configured in acommon package. Indeed, any or all of the various components of amodule, whether control logic or other components, can be combined in asingle package or separately maintained and can further be distributedin multiple groupings or packages or across multiple locations.

Additionally, the various embodiments set forth herein are described interms of exemplary block diagrams, flow charts and other illustrations.As will become apparent to one of ordinary skill in the art afterreading this document, the illustrated embodiments and their variousalternatives can be implemented without confinement to the illustratedexamples. For example, block diagrams and their accompanying descriptionshould not be construed as mandating a particular architecture orconfiguration.

While various embodiments of the present disclosure have been describedabove, it should be understood that they have been presented by way ofexample only, and not of limitation. Likewise, the various diagrams maydepict an example architectural or other configuration for thedisclosure, which is done to aid in understanding the features andfunctionality that can be included in the disclosure. The disclosure isnot restricted to the illustrated example architectures orconfigurations, but the desired features can be implemented using avariety of alternative architectures and configurations. Indeed, it willbe apparent to one of skill in the art how alternative functional,logical or physical partitioning and configurations can be implementedto implement the desired features of the present disclosure. Also, amultitude of different constituent module names other than thosedepicted herein can be applied to the various partitions. Additionally,with regard to flow diagrams, operational descriptions and methodclaims, the order in which the steps are presented herein shall notmandate that various embodiments be implemented to perform the recitedfunctionality in the same order unless the context dictates otherwise.

Although the disclosure is described above in terms of various exemplaryembodiments and implementations, it should be understood that thevarious features, aspects and functionality described in one or more ofthe individual embodiments are not limited in their applicability to theparticular embodiment with which they are described, but instead can beapplied, alone or in various combinations, to one or more of the otherembodiments of the disclosure, whether or not such embodiments aredescribed and whether or not such features are presented as being a partof a described embodiment. Thus, the breadth and scope of the presentdisclosure should not be limited by any of the above-described exemplaryembodiments.

Terms and phrases used in this document, and variations thereof, unlessotherwise expressly stated, should be construed as open ended as opposedto limiting. As examples of the foregoing: the term “including” shouldbe read as meaning “including, without limitation” or the like; the term“example” is used to provide exemplary instances of the item indiscussion, not an exhaustive or limiting list thereof; the terms “a” or“an” should be read as meaning “at least one,” “one or more” or thelike; and adjectives such as “conventional,” “traditional,” “normal,”“standard,” “known” and terms of similar meaning should not be construedas limiting the item described to a given time period or to an itemavailable as of a given time, but instead should be read to encompassconventional, traditional, normal, or standard technologies that may beavailable or known now or at any time in the future. Likewise, wherethis document refers to technologies that would be apparent or known toone of ordinary skill in the art, such technologies encompass thoseapparent or known to the skilled artisan now or at any time in thefuture.

The presence of broadening words and phrases such as “one or more,” “atleast,” “but not limited to” or other like phrases in some instancesshall not be read to mean that the narrower case is intended or requiredin instances where such broadening phrases may be absent. The use of theterm “module” does not imply that the components or functionalitydescribed or claimed as part of the module are all configured in acommon package. Indeed, any or all of the various components of amodule, whether control logic or other components, can be combined in asingle package or separately maintained and can further be distributedin multiple groupings or packages or across multiple locations.

Additionally, the various embodiments set forth herein are described interms of exemplary block diagrams, flow charts and other illustrations.As will become apparent to one of ordinary skill in the art afterreading this document, the illustrated embodiments and their variousalternatives can be implemented without confinement to the illustratedexamples. For example, block diagrams and their accompanying descriptionshould not be construed as mandating a particular architecture orconfiguration.

1. A method comprising: receiving an encoded video; determining asaliency-weighted image quality metric for each video frame of theencoded video; determining video frames of the encoded video with asaliency-weighted image quality metric that is below a threshold; andassembling a playlist including the video frames of the encoded videowith a saliency-weighted image quality metric that is below thethreshold.
 2. The method of claim 1, wherein determining asaliency-weighted image quality metric for each video frame of theencoded video comprises: determining a per-pixel image quality vector ofthe video frame; determining per-pixel saliency values of the videoframe; and computing the saliency-weighted image quality metric of thevideo frame based on the per-pixel image quality vector and theper-pixel saliency values.
 3. The method of claim 2, wherein thesaliency-weighted image quality metric of each video frame is determinedbased on the equation${\hat{q} = {\frac{1}{N}\left( {q \cdot s^{w}} \right)}},$ wherein{circumflex over (q)} is the metric, (·) denotes the inner product, q isthe per-pixel image quality vector of the video frame, s is a vector ofthe per-pixel saliency values of the video frame, w is a weight given tothe vectors, and N is the number of pixels in the video frame.
 4. Themethod of claim 2, wherein determining a per-pixel image quality vectorof each video frame comprises comparing the video frame with a referencevideo frame.
 5. The method of claim 2, further comprising: displayingcolor-coded per-pixel visible difference visualization between theencoded video and a reference video.
 6. The method of claim 5, whereinthe color-coded per-pixel visible difference visualization is displayedfor a video frame of the encoded video using at least the per-pixelimage quality vector and the per-pixel saliency values determined forthe video frame.
 7. The method of claim 5, further comprising: providinga graphical user interface including one or more controls for selectingone of a plurality of video quality assessment methods to display thecolor-coded per-pixel visible difference visualization.
 8. The method ofclaim 5, wherein the color-coded per pixel visible differencevisualization displays no color to denote no visible artifacts and colorto denote visible artifacts.
 9. The method of claim 1, furthercomprising: displaying a quality prediction plot of the encoded videothat provides video quality normalized as a function of time code orvideo frame number of the encoded video.
 10. The method of claim 1,wherein assembling the playlist comprises: marking time codes or framenumbers of the video frames of the encoded video with asaliency-weighted image quality metric below the threshold; andassembling the marked time codes or frame numbers into the playlist. 11.A system, comprising: a non-transitory computer-readable mediumoperatively coupled to a processor and having instructions storedthereon that, when executed by the processor, cause the system to:receive an encoded video; determine a saliency-weighted image qualitymetric for each video frame of the encoded video; determine video framesof the encoded video with a saliency-weighted image quality metric thatis below a threshold; and assemble a playlist including the video framesof the encoded video with a saliency-weighted image quality metric thatis below the threshold.
 12. The system of claim 11, wherein determininga saliency-weighted image quality metric for each video frame of theencoded video comprises: determining a per-pixel image quality vector ofthe video frame; determining per-pixel saliency values of the videoframe; and computing the saliency-weighted image quality metric of thevideo frame based on the per-pixel image quality vector and theper-pixel saliency values.
 13. The system of claim 12, wherein thesaliency-weighted image quality metric of each video frame is determinedbased on the equation${\hat{q} = {\frac{1}{N}\left( {q \cdot s^{w}} \right)}},$ wherein{circumflex over (q)} is the metric, (·) denotes the inner product, q isthe per-pixel image quality vector of the video frame, s is a vector ofthe per-pixel saliency values of the video frame, w is a weight given tothe vector s, and N is the number of pixels in the video frame.
 14. Thesystem of claim 12, wherein determining a per-pixel image quality vectorof each video frame comprises comparing the video frame with a referencevideo frame.
 15. The system of claim 12, wherein the instructions, whenexecuted by the processor, further cause the system to: displaycolor-coded per-pixel visible difference visualization between theencoded video and a reference video.
 16. The system of claim 15, whereinthe color-coded per-pixel visible difference visualization is displayedfor a video frame of the encoded video using at least the per-pixelimage quality vector and the per-pixel saliency values determined forthe video frame.
 17. The system of claim 15, wherein the instructions,when executed by the processor, further cause the system to: provide agraphical user interface including one or more controls for selectingone of a plurality of video quality assessment methods to display thecolor-coded per-pixel visible difference visualization.
 18. The systemof claim 15, wherein the color-coded per pixel visible differencevisualization displays no color to denote no visible artifacts and colorto denote visible artifacts.
 19. The system of claim 11, wherein theinstructions, when executed by the processor, further cause the systemto: display a quality prediction plot of the encoded video that providesvideo quality normalized as a function of time code or video framenumber of the encoded video.
 20. The system of claim 11, whereinassembling the playlist comprises: marking time codes or frame numbersof the video frames of the encoded video with a saliency-weighted imagequality metric below the threshold; and assembling the marked time codesor frame numbers into the playlist.